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ABSTRACT 


This thesis reviewed two different approaches to benefit analysis, 
benefit comparison and user satisfaction, that could be applied to the 
evaluation of proposed information systems which are under 
consideration for acquisition by the federal government. Currently the 
General Services Administration only recommends that present value 
analysis methods be used in the analysis of alternatives even though 
the GSA specifies that non-cost factors should be evaluated in such an 
analysis. 

Different benefit comparison and user satisfaction methods are 
reviewed for their particular advantages and disadvantages. A 
discussion is given on how selected methods of each approach may be 
used within the federal government for the evaluation of alternatives. 
Suggestions are made for ways of conducting a more complete analysis 
of alternatives through incorporating present value analysis, benefit 
comparison of non-cost factors, and analysis of user satisfaction, into 


one comprehensive analysis. 
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I. INTRODUCTION 


A. BACKGROUND 

The need for benefit analysis methods capable of being 
applied to the evaluation of proposed information systems 
has long been recognized. Twenty years ago Chervany and 
Dickson (Chervany and Dickson,1970) asserted then that it 
was time to bemeen uate the decision-making process 
involving the development of information system 
alternatives. In the mean time the demand for information 
systems, and hence the demand for benefit analysis of these 
systems, has grown substantially. Abdel-Hamid and Madnick 
(Abdel-Hamid and Madnick,1989) estimate the increase in 
demand at a hundredfold over the last two decades. Kitfield 
(Kitfield, 1989) estimated the demand for avionic software 
was increasing 25 percent per year. This trend may be even 
more true for the Department of Defense (DoD) as one expert 
has described what he calls the "military software crisis" 
(Kitfield,1989). Kitfield describes this crisis as, 
"runaway demand and a profound shortage of software 
programmers” (Kitfield,1989). The problem of selecting an 
information system alternative may now be focused more on 
which alternative to invest in first. This situation 
reinforces the need for benefit analysis to prioritize 


software projects. 


B. OBJECTIVES 

The objective of this thesis, is to determine what 
methodologies can be used to analyze the benefits to be 
gained from a proposed information system. More 
specifically the objective is to examine what current 
benefit analysis methods can be used in the evaluation of 
different competitive bids submitted in response to a 
solicitation from the federal government for a proposed 
information system. Of special interest is how intangible 
benefits of advanced information technology are treated in 


such an analysis. 


C. RESEARCH QUESTIONS 

The research concerns of this thesis are really two 
fold. First, which benefit analysis methods can be employed 
to compare the benefits of two competing proposed 
information systems to determine the alternative that is the 
most advantageous to the government. Secondly, which 
benefit analysis methods are capable of calculating a 
monetary value of the benefits from a complex information 


system prior to that system's development. 


D. SCOPE AND LIMITATIONS 

This study will focus on benefit analysis methods that 
can be used in the evaluation of alternatives which 1s 
conducted in the process of a federal acquisition of an 


information system. Benefit analysis methods will also be 


examined that can be incorporated into a larger cost-benefit 
analysis. The subject of risk assessment, in relation to 
information systems, is included under benefit analysis of 


proposed information systems for the purpose of this thesis. 


Il. METHODOLOGY 


A. SELECTING STUDIES OF BENEFIT ANALYSIS 

1. Choosing a Medium from which to Select Studies 

Published articles were chosen as the source of 

studies on benefit analysis in both existing and proposed 
information systems. Studies on existing systemn wee 
included to gain a better insight into the subject of 
benefit analysis of proposed information systems. Only 
articles of benefit analysis in English-language journals, 
in the field of management information systems and the 
related field of management science, were surveyed. 

2. Choosing a Time Span of the Studies to Survey 

The work on this literature review began in June, 

1990 and was completed in January, 1991. The last articles 
that were excepted for inclusion were published in the 
calendar year of 1990. The year 1970 was selected as the 
starting point for the bibliographical search of this 
literature review. However, a time limit was not placed on 
articles or supporting publications that could be 
incorporated into the research based on references in 
articles found in the main bibliographical search or books 


that supported the underlying concepts of an article. 


3. Search for Key Words 
a. The Initial Bibliographical Search 
An initial search was conducted manually of 
Harvard Business Review and Management Information Systems 
Quarterly to find key words that could be used in the search 
of on-line bibliographic data. These journals were used 
because of their history of articles published in the field 
of management information systems. The table of contents of 
every issue between 1970 and 1990 was manually reviewed. If 
a title appeared relevant to the subject of benefit analysis 
of information systems then the article was reviewed. After 
reading the article and determining that indeed it did 
pertain to the subject, the article was then copied and 
saved for later referral. 
b. Key Words Found 
The search for key words identified 19 articles 

from these two journals. From the initial search it could 
also be concluded that the authors of the 19 articles chose 
diverse key words to describe their articles. Key words 
included: 

- Cost/Benefit Analysis 

- Cost/Benefit Calculations 

- Information Economics 

- Information System Investment 

- Information Technology Investment 


- Value Analysis 


As the dissimilarity of key words found in the initial 
search was discovered, it was determined that a single on- 
line search with a single set of key words would not 
sufficiently identify the number of articles desired on the 
subject of benefit analysis of information systems. The 
problem was that the key words that were found were just to 
diverse for the on-line system to be effective. 
4. Manual Bibliographical Search 
As a result of the difficulties with key words, a 

manual bibliographical search was undertaken as opposed to 
an on-line search. This main search was conducted in the 
same manner as the preliminary search. A list of eight 
additional journals was gathered based on their reputation 
for publishing studies in the field of management 
information systems. The journals searched manually were: 

- Communications of the ACM * 

- Data Base 

- Decision Sciences 

- Harvard Business Review 

- Information and Management 

- Journal of Management Information Systems 

- Management Science 

- Management Information Systems Quarterly 

- Omega 

- Sloan Management Review 


* ACM: Association for Computing Machinery 


a. Conduct of the Manual Search 

Once the list of journals was established the 
search was conducted systematically. Journal by journal, 
the table of contents for each issue available, during the 
time period of 1970 to 1990, was examined for a possible 
article relevant to the subject of interest. Articles with 
fitting titles were copied for latter reference if after 
being reviewed it was concluded that they were appropriate. 

b. Other Sources of Literature 

Other sources of reference were considered to 
clarify certain points brought up in the articles that had 
been initially collected and reviewed. The references of 
the articles located in the main search were used to 
identify further articles that could pertain to the subject 
of this study. These sources were located through on-line 
bibliographical searches by either author, title, or 
subject. The on-line search was useful to locate these 
additional sources within the library for in this instance 
very specific: author names, and titles, were used to query 
the on-line system for exact locations of individual works. 
A final source of literature that comprises the foundation 
of this thesis came from the courses that this author was 
exposed to at the Naval Postgraduate School. A 
comprehensive inventory of the literature found and used for 


this thesis is presented in the bibliography. 


B. SORTING THE WORKS ON BENEFIT ANALYSIS 

In combination, the preceding search methods yielded a 
total of 50 references that were considered for possible 
contribution to the study of benefit analysis of proposed 
information systems. These references were organized into 
three groups: benefit comparison, user satisfaction, and 
miscellaneous literature. 

1. Benefit Comparison 

The articles in this group are distinguished from 

the others by the condition that they present methods of 
analyzing benefits by assessing various system attributes. 
The methods discussed under benefit comparison are: the 
additive weight model, the eigenvector model, and the multi- 
attribute utility model. Chapter IV is dedicated to benefit 
comparison. 

2. User Satisfaction 

The references in this group, in contrast to the 

groups previously mentioned, propose to analyze benefits not 
through an objective indicator, but through a survey to 
determine the subjective satisfaction that users have with 
their respective information system. The procedures of this 
method incorporate various formulas and models to interpret 
the responses of user satisfaction questionnaires. User 


satisfaction is the subject of Chapter V. 


3. Miscellaneous Literature 

The last group that references were organized into 
was miscellaneous. This group contained articles on the 
importance of information systems and their predominance 
throughout the business world and DoD. Articles were also 
placed in this group that stressed the value of information 
and the need for benefit analysis of information systems. 
Articles on the competitive advantage to be gained from 
information systems were also placed in the miscellaneous 
group. In spite of the expressed need for benefit analysis, 
the group of miscellaneous literature did not contribute to 
any specific benefit analysis technique. As such, the 
references in this group were largely used as background 
material that stressed the importance of information system 
benefit analysis and that put benefit analysis in 
perspective in relation to analysis of alternatives and the 


rest of the development life cycle for information systems. 


III. BACKGROUND 


A. GUIDELINES FOR FEDERAL ADP ACQUISITIONS 

The Brooks Act, public law 89-306, established the U.S 
General Services Administration (GSA) as the pee eunenenn 
authority for ADP equipment acquisitions within the federal 
government. The Act instructs the GSA to "provide for the 
economic and efficient purchase, lease, and maintenance of 
automated data processing equipment by federal agencies" 
(U.S. GSA,1990). This authority was increased through the 
enactment of public law 99-500 to also include the federal 
acquisition of such things as; software, firmware, and 
computer related contracted services to include programming 
and support services. In fulfillment of their 
responsibility the GSA publishes both procedural guides, 
such as A Guide for Requirements Analysis and Analysis of 
Alternatives, and the federal information resources 
management regulation (FIRMR). (U.S. GSA,1990) 

The Warner Amendment, as enacted under public law 99- 
500, did release DoD from the jurisdiction of the Brooks Act 
for certain ADP functions that dealt with national security 
or military operations. However, the Warner Amendment did 
not release DoD from the conditions of the Brooks Act for 
ADP acquisitions that deal with a vast number of other 


functions such as logistics and administrative activities. 
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As such the DoD must abide by FIRMR when making a great 


number of ADP and information system acquisitions. 


B. GSA POLICY FOR THE ANALYSIS OF ALTERNATIVES 
In the process of making a federal acquisition of ADP 

equipment or systems the GSA requires that an analysis of 
alternatives be performed. The objective of this analysis 
is to determine the most advantageous alternative to the 
government. As a part of this analysis of alternatives the 
GSA has mandated that both cost and non-cost factors are to 
be evaluated. The specific cost and non-cost factors that 
should be considered in this type of analysis have been 
determined by the GSA and are listed, and described, in A 
Guide for Requirements Analysis and Analysis of Alternatives 
(U.S. GSA,1990). The cost factors are: 

- conversion 

- personnel 

- supplies 

- energy 

- maintenance 

- space 

- administrative costs of contracting 

- contract price 
The cost factors above are all quite tangible and relatively 
easy to evaluate in the course of an analysis of 
alternatives. In fact the GSA has recommended three 
different methods for use in the analysis of cost factors: 


Jka 


present value analysis, net present value, and benefit-cost 


ratio. (U.S. GSA,1990) 


C. ANALYSIS OF COST FACTORS 
1. Description 

Of the techniques recommended by GSA for the 
analysis of cost factors, present value analysis 1S common 
to capital investment and calculates the present worth of a 
monetary amount to be received in the future. Present value 
analysis takes into account the time value of money, or the 
concept that a dollar is worth more today than a year from 
today. There are several different approaches to present 
value analysis to included: present value of a single future 
payment, present value net benefits, and present value for 
competing alternatives. 

a. Present Value of a Single Future Payment 

In Garrison (1988), present value analysis is 

described in the context of a the receipt of a single 
amount, or benefit, in the future. The formula for 


calculating present value of a one time payment in the 


future is given in Formula l. (Garrison,1988) 
pea 
(1+r)” (1) 


In Formula l, P represents the present value of 


the benefit in monetary terms. F is the monetary value of 


12 


the benefit to be received in the future. ris the interest 
rate and n is the number of years in the future that the 
benefit will be received. (Garrison,1988) 

b. Present Value Net Benefits 

Thompson (1980) examines present value analysis 

in terms of discovering the sum of all the systems results, 
both good and bad, or in other words both liabilities and 
assets. In his work Thompson uses the expression "present- 
valued compensating variation” when referring to the 
difference between the cost and benefit of a particular 


attribute. Thompson's formula for present-valued net 


benefits is given in Formula 2. (Thompson,1980) 
CV. 
p= ij 
p> > (1+d)” 2) 


In Formula 2, CV; ; is the compensating variation 
for benefit j on the user i. dis the discount rate. y is 
the number of years until the user i is affected by benefit 
j.- mis the number of different benefits. nis the number 
of users benefitted by the system. (Thompson,1980) 

Thompson's present-valued net benefits method 
should not be confused with the net present value (NPV) 
method. As recommended by GSA, the NPV is equal to 
difference between the total present value of all of the 


benefits and the total present value of all of the costs. 


HS 


NPV, unlike present-valued net benefits, does not attempt to 
sum the differences between costs and benefits of each 
characteristic of an information system. (U.S. GSA,1990) 

c. Present Value for Competing Alternatives 

In NAVDAC PUB 15 (Dec,1980) present value 
analysis is used to evaluate alternatives based on lowest 
cost. In this application of present value analysis, the 
benefits of all alternatives being considered must be equal 
and a selection is made by determining the lowest present 
value of the costs of the different alternatives. Also, by 
this method, the service lives of the alternatives must be 
finite and equal among alternatives, or be placed in equal 
terms. In computing present values the NAVDAC method uses 
standard tables of discount factors in which the present 
value of a dollar can be found for both a single cash flow 
and cumulative uniform cash flows. Uniform here being the 
same amount paid every year. This form of the present value 
may be easier to conceptualized if placed in the context of 
hardware acquisitions. (NAVDAC PUB 15,1980) 
2. Example of Present Value Analysis 

In this example two high capacity printers are under 
consideration for purchase. Both of the printers have the 
same exact capabilities, such as the same output speeds, 
font styles, and buffer sizes. One of the printers, 
alternative A, has a service life of six years. The other, 


alternative B, has a service life of three years. In this 
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Situation the NAVDAC method of present value analysis can be 
applied for the benefits are equal and the service lives can 
be placed in equal terms. (NAVDAC PUB 15,1980) 

To further define the example, alternative A costs 
$10,000 to purchase and has annual reoccurring cost of 
$4,000. Alternative B costs $8,000 to purchase and has 
annual reoccurring costs of $5,000. Both alternatives are 
not expected to retain any value at the end of the economic 
lives. (NAVDAC PUB 15,1980) 

a. Present Value Using a S1x Year Service Life 

The present value for the two alternatives using 
a six year service life would be found as follows. For 
alternative A the purchase price would be added to the 
reoccurring expenses multiplied by the factor for a six year 
cumulative uniform series. In alternative B two printers 
have to be purchased so that their combined service life 
equals six years. The present value of alternative B is the 
purchase price for the first printer added to the 
reoccurring expenses multiplied by the factor for a six year 
cumulative uniform series and then added to the purchase 
price of the second printer multiplied by the factor for a 
Single payment in three years. Algebraically the present 
values determination of the alternatives is shown below. 
(NAVDAC PUB 15,1980) 


A 


$10,000 + ($4,000 x 4.570) = $28,280 


B $8,000 + ($5,000 x 4.570) + ($8,000 x .788)=$37,154 
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b. Present Value Using a Three Year Service Life 
Had a three year service life been used in the 

comparison only one purchase in alternative B would have to 
have been considered. However, as alternative A would have 
half of its service life remaining at the end of three 
years, then half of its purchase price, multiplied by the 
factor for a single cash flow in three years, would have to 
be deducted from the calculation to represent the remaining 
value. Also, the reoccurring expenses multiplied by the 
factor for a three year cumulative uniform series, not a Six 
year series. Algebraically the present values determination 
of the alternatives using this three year service life is 
given below. (NAVDAC PUB 15,1980) 


A 


$10,000 + ($4,000 x 2.609) - ($5,000 x .788)=$16,496 


B 


$8,000 + ($5,000 x 2.609) = $21,045 

3. Advantages & Disadvantages of Present Value Analysis 

A key advantage of the present value technique is 

that organizations are familiar with it for the technique is 
used to make other capital budgeting decisions 
(Garrison,1988). Another advantage is that present value 
analysis is a simple means of comparing alternatives when 
the benefits and project lives are the same (NAVDAC PUB 
15,1980). Then possibly the most compelling advantage of 
present value analysis is that it is one of the three cost 
analysis methods recommended by the GSA for use in the 


analysis of alternatives. This analysis being required for 
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each federal acquisition of ADP equipment or information 
systems. 

Present value analysis alone is seen by some as ill 
suited for the analysis of some types of information 
systems, such as decision support systems (Keen,1981). The 
technique is not recommended in this situation for it is 
difficult to quantify the benefits of such a system 
capabilities such as AD-HOC analysis (Keen,1981). Present 
value analysis, as described above, is also criticized for 
it does not take into account risk or the probability of the 
benefit occurring (Couger et al,1982). Possibly for these, 
Or similar reasons GSA has also required that non-cost 


factors be included in the analysis of alternatives. 


D. ANALYSIS OF NON-COST FACTORS 
The non-cost factors that the GSA has specified that are 

to be included in an analysis of alternatives have been 
separated by GSA into two groups; functional factors and 
risk factors. The functional factors include: (U.S. 
GSA,1990) 

- obsolescence 

- availability 

- reliability 

- maintainability 

- expandability 

- flexibility 

- security 
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- privacy 

- personnel impacts 

- user acceptance 

- accountability 

The risk factors are: (U.S. GSA,1990) 

- financial 

- technical 

- schedule 

The inclusion of these functional and risk factors into 

an analysis of alternatives offsets the disadvantages of 
conducting an analysis solely on the basis of cost factors. 
Those disadvantages namely being; the lack of addressing 
risk, and the inappropriate nature of attempting to apply 
cost-benefit analysis methods to benefits that are difficult 
to quantify in dollar terms. As such, there should be a 
clear advantage of conducting an analysis of alternatives 
that incorporates both cost and non-cost factors. However, 
even though GSA does directs that the functional and risk 
factors be included in the analysis of alternatives, the GSA 
guides do not recommend or mension any non-cost analysis 
method which should be used in such an analysis. (U.S. 


GSA,1990) 
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IV. ANALYSIS METHODS FOR NON-COST FACTORS 


There are existing benefit comparison methods that may 
well suited for fulfilling the GSA requirement for non-cost 
analysis. That analysis, of course, being part of an 
analysis of alternatives leading up to the federal 
acquisition of ADP equipment or information systems. These 
benefit comparison methods not only are useful in 
contrasting the benefits of the various alternatives, they 
can also be used for making a selection from proposed 
alternatives based on non-cost factors. The non-cost 
factors in this context may be described as attributes of 
information systems and may be analyzed as well as measured 
as such by the different benefit analysis methods. The 
supporters of objective quantification methods believe that 
analysts, who describe the benefits of information systems 
as completely intangible, are not putting forth the effort 
to find measures for the benefits (Couger et al,1982). The 
benefit comparison methods that will be addressed are: the 
additive weight model, the eigenvector model, and the multi- 


attribute utility model. 


A. THE ADDITIVE WEIGHT MODEL 
l. Description 
The additive weight model consists of three steps; 
determining weights, scoring alternatives, and selecting the 
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best alternative. Prior to using the model however, the 
benefits must be defined and broken down into the separate 
attributes or characteristics of the information system. 
Such attributes could include: memory capacity, calculation 
speed, or the manufacturer's reliability. These attributes 
can be divided by categories, such as hardware and software, 
or listed together in one group. The attributes are not 
required to be independent of each other (Shoval and 
Lugasi,1987). Defining the attributes establishes common 
references in which the additive weight model can be 
implemented by analysts and decision makers alike. (Shoval 
and Lugasi,1987) 
a. The Three Steps of the Additive Weight Model 

The first step determines attribute weights for 
each attribute. These may be obtained through point 
allocation where a decision maker assigns a number between 0 
and 1 to each attribute to reflect its relative importance 
(Schoemaker and Waid,1982). The sum of all the attribute 
importance weights must equal l. If attributes are divided 
into categories then category importance weights must also 
be selected and their sum must also equal 1l. 

The second step, scores alternatives, by having 
the decision maker evaluate each alternative and assess a 
rating for each attribute. This is done one attribute at a 
time and only the attributes receive a rating, not the 


categories or the overall system. The final step, selection 
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of the best alternative, is made by choosing the information 
system that maximizes the summation of attribute scores 
multiplied by the respective attribute and category weights. 
This step is expressed in Formula 3 and demonstrated in 
Table 3. (Shoval and Lugasi,1987) 
b. The Additive Weight Formula 

Formula 3 is the additive weight formula. When 
using this formula four conditions must hold true. First, 
the sum of the category weights must equal 1. Second, 
attribute weights range from 0 to l. Third, attribute 
scores must be less than or equal to l. Finally, the sum of 
the attribute importance weights must equal 1. (Shoval and 
Lugasi ,1987 ) 

The following notation is used in Formula 3 and 
Tables 1 through 3. Z,; expresses the category importance 
weights of m categories. W, represents attribute importance 
weights of n attributes. Finally Vij symbolizes the ith 


attribute score for alternative j. (Shoval and Lugasi,1987) 


m nm), 


se 0 ae (3) 


2. Example of the Additive Weight Model 
In Shoval and Lugasi (1987) an example of the 
additive weight model is given where there are four 


alternatives to select from and 14 attributes to be 


21 


evaluated which were split into three categories. The four 
proposed systems are alternatives A through D. The three 
categories are hardware, software, and support. The 
attributes, listed by category, are given in Table 1 along 
with their respective importance weights. Alternative 
scores by attribute are given in Table 2. (Shoval and 


Lugasi ,1987) 


Table 1. Category and Attribute Importance Weights 


Hardware 

X1 Memory Capacity 

X2 Calculation Speed 

X3 I/O Speed 

X4 Equipment Reliability 

X5 Flexibility for Expansion 
Software 

X6 Availability for Scientific Software 
X7 Flexibility to Changes 

X8 Software Performance 
Support 

X9 Manufacturer's Reliability 
X10 Supplier's Reliability 

X11 Supply Time 

X12 Quality of Hardware Support 
X13 Quality of Software Support 
X14 Quality of Documentation 
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Table 2. Alternative Scores by Attribute 


Alternatives 


Attributes 


HMOrMOOrFOFOrFrFrOOO 
OWOWWOWOWNDOBD®D 
HMOrMOdOrFOrFrOOFrO0CO 
OW OWOOWOWDWOWDAD 
MR OF OOOrFOCOOCOOC Oo 
Ooo wmo~ DDO WDO~1WAWO® 
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Table 3. Calculation of the Weighted Score for 
Alternative A 


Attribute Category x Attribute x Attribute = Weighted 
Weight Weight Score Score 


© 
Le. °) 


Ow Owwdsd wow O® oO ® 


0.3 
0.3 
0.3 
0.3 
0.3 
0.3 
0.3 
0.3 
0.4 
0.4 
0.4 
0.4 
0.4 
0.4 
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X14 





Total Weighted 
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From Formula 3 and Tables 1 through 3, alternative 
A, with a score of 0.9158, has the highest total weighted 
score of the four alternatives. Alternative A would thus be 
the information system selected by the additive weight 
model. The weighted scores of alternatives B, C, and D are 
0.9024, 0.8399, and 0.8557 respectively. (Shoval and 
Lugasi ,1987) 

3. Advantages and Disadvantages of Additive Weight 

Besides being simple in nature the additive weight 
model also has the advantage that sensitivity analysis, or 
receptiveness to change, can be measured for the weights 
assigned to the various attributes and categories. The 
disadvantage is that this model cannot check the consistency 
of evaluators; nor does it consider risk or uncertainty. 
Assigning weights and scoring alternatives is also a 
subjective appraisal of information system benefits. 


(Shoval and Lugasi,1987) 


B. THE EIGENVECTOR MODEL 

1. Description 

Building on the additive weight model the 

eigenvector model provides a means for determining the 
attribute and category weights as well as scoring the 
alternatives. Final system selection in the eigenvector 
model is made using Formula 3 of the additive weight model. 
In the eigenvector model attribute weights are found by a 
process which starts with pairwise comparisons. A decision 
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maker completes a matrix in which both the number of columns 
and rows equal the number of attributes being evaluated. 

For instance, if four attributes were being examined, then a 
4x4 matrix would be required. Each cell of the matrix 
represents a comparison between two of the attributes. A 
nine-point scale is used in the model with 1 representing 
equality between attributes and 9 depicting absolute 
preference of one attribute over the other. In the matrix 
in Figure 1, attribute xl is favored 3/2 times to attribute 
x5. The consistency of the decision maker can be verified 
through an inspection of inverse relationships. Here x5 is 
favored 2/3 times to xl so the decision maker is being 


consistent. (Shoval and Lugasi,1987 ) 


Figure 1. Sample Matrix 





The eigenvector of this matrix is then computed for 
the maximum eigenvalue. This eigenvector is then normalized 
so that the sum of the elements in the vector equals l. 
Computing and normalizing these eigenvectors can be done 
through the use of a computer package such as EISPACK, which 
is a package of fortran IV programs (Goos and 
Hartmanis,1977). The elements in this final form of the 
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eigenvector are the respective attribute weights. Category 
weights are then determined in a similar fashion with the 
dimensions of the matrix equal to the number of categories. 
(Shoval and Lugasi,1987) 

To score the alternatives by attribute, a matrix of 
pairwise comparisons is used. This time the number of 
matrices used 1s equivalent to the number of attributes. 

The dimensions of each of these matrices is set to the 
number of alternatives. The elements of the normalized 
eigenvector are the solution. Here they represent the 
scores of the alternatives for the respective attributes. 
(Shoval and Lugasi,1987) 

2. Example of the Eigenvector Model 

The situation used in this example is the same as 
that used above for the additive weight model: four 
alternatives, three categories, and 14 attributes. The four 
alternatives are still labeled A through D and the 
categories and attributes remain the same from Table 1 
above. In Figure 2 of Appendix A the matrices for 
determining attribute weights are displayed. Figure 3 of 
Appendix A is the matrix of pairwise comparisons of the 
three categories. In Figure 4 of Appendix A the 14 matrices 
are presented that were used to define the alternative 
scores by attribute. Finally, Table 4 contains the category 
and attribute weights, and lists the weighted scores for the 


each alternatives which is used to produce their respective 
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ranking as determined by the eigenvector model. (Shoval and 


Lugasi,1987) 
Table 4. Scores and Ranking of Alternatives 


Alternatives 
Attribute Weight A B Cc 


Pee 
.244 ay AP 
-244 . 286 
.O91 Pa Bl 
2250 - 316 
- 169 - 286 
7333 
. 400 . 286 
200 
- 400 
1333 
SoZ 
72S 
.062 
. £S9 
-139 
.072 


Weighted Score 
Ranking 





3. Advantages & Disadvantages of the Eigenvector Model 
The advantage of the eigenvector model is that it 

facilitates a review of a decision makers consistency 
throughout the analysis by conducting an inspection of 
inverse relationships between respective paired comparisons. 
However, the eigenvector model ignores attribute 
interdependence and does not reflect risk or uncertainty 
involved with the various alternatives in the analysis. In 
the course of a large scale analysis pairwise comparisons 
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can also become quite difficult when the number of pairs is 


increased to a large number. (Shoval and Lugasi,1987) 


C. THE MULTI-ATTRIBUTE UTILITY MODEL 
1. Description 

Like the additive weight and eigenvector models the 
multi-attribute utility model requires a list of applicable 
attributes to be evaluated. While benefits of information 
systems are not unidimensional, it is probable that 
dimensionality can be contained or at least reduced by 
including only the most relevant attributes to the systems 
being evaluated (Ahituv,1980). The multi-attribute utility 
model itself has two variations, an additive model and a 
multiplicative model (Formulas 4 and 5 respectively). The 
variants differ in the way they treat risk. (Shoval and 
Lugasi ,1987) 


a. Multi-Attribute Otility Formulas 


n 
U;= 0 kU; (4) 
1=1 
n 
( (1+kk,U,,))-1 
” &, iV4j (5) 
J k 


In both Formulas 4 and 5, U; depicts the utility 
of alternative j. U;; refers to the utility of attribute i 
in alternative j and U itself is constrained to be between 0 


and 1 by the constant k. Formula 6 below is used to 
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determine k. k; here portrays the weight of attribute i. 


(Shoval and Lugasi,1987) 


fl 
1+k=]] (1+kk,) (6) 
1=1 


b. Model Selection 

The use of either the additive or multiplicative 
model is dependent on the decision maker's outlook on risk. 
The decision aid for that determination is the summation of 
all the attribute weights. If this sum is equal to l, then 
the decision maker is indifferent to risk and the additive 
model is selected. If the sum is greater than l, then the 
multiplicative model is selected and k set between O and -l. 
When the summation of attribute weights is less than l, then 
the multiplicative model is also used, yet in this case k is 
set to a value greater than 0. (Shoval and Lugasi,1987) 

c. Application of the Multi-Attribute Utility Model 

The multi-attribute utility model requires both 
utility and preference independence of the various 
attributes. Once the attribute independence has been 
confirmed the evaluation of the utility function for every 
attribute can then be based on two axioms, transitivity and 
continuity. The transitivity axiom states that if 
alternative a; is favored over aj, and a; is favored over 4a,, 
then alternative a, is superior to alternative a,. The 


l 
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continuity axiom refers to the state that if alternative a; 


is favored over aj. and a; is favored over a,, then a gamble 


on 4a; and a, can be performed so that there will be a point 
where the decision maker will be indifferent between 
choosing the gamble or being assured of receiving 
alternative aj: If the axioms can be met then the utility 
function can be found through the multi-attribute utility 
model. (Shoval and Lugasi,1987) 
d. Forming the Utility Function 

To form the utility function for each attribute, 
with the above axioms previously being met, the two extreme 
points of the utility curve must be defined. The highest 
level for attribute i is designated as i* and u(i*) is set 
to equal 1. The lowest level of attribute i is represented 
by i! and u(i!) is set to 0. i is designated as i' when 
conducting the gambling technique in evaluating the utility 
of a particular level of i. The decision maker is then 
presented with the following situation. Given two 
alternatives, A and B, with all attributes between the two 
alternatives, other than i, being equal. In alternative A 
there is a 100% probability that attribute i will be i'. 
Alternative B is the gamble with a probability of p that 
attribute i will be i* and a probability of 1-p that i will 


be i?, The decision maker is then asked to determine at 


ee 


what level of p he would be indifferent between the two 


alternatives. When the decision maker is indifferent to the 
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two alternatives the utility of alternative A equals the 
expected utility of alternative B, or in other words U(i') = 
pu(i'*) ei (1-p)u(i®). The value of U(i') is then known to be 
p, for u(i*) is 1 and u(i’) is 0. Other utility points for 
attribute i are found in the same manner by varying the 
level of i‘. The utility curve for i is formed by 
connecting the utility points. A continuous curve is not 
required for it is suitable to evaluate only a range of 
relevant values for each attribute. (Shoval and 
Lugasi,1987 ) 

e. Determining the Attribute Weights 


The attribute weights, k;, are found in the same 


1’ 
way that the attribute utility points were found. To 
define the attribute weights the decision maker is given a 
Situation with two alternatives, A and B. This time 
alternative A has attribute i at the highest level, or a 
and the remaining attributes at their lowest level, or i?, 
Alternative B iS again a gamble with a probability of p that 
all of the attributes are at their respective highest level, 
and a probability of l-p that all of the attributes are at 
their lowest level. The decision maker is then asked to 
choose between the two alternatives by answering the 
following question: "At what level of p would you be 
indifferent between the two alternatives?" The value of k; 


is then known to be p by the same reasoning that U(i') is 


equal to p. With the utility function and weights found for 
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every attribute the appropriate version of the model can 
then be applied. (Shoval and Lugasi,1987) 
2. Example of the Multi-Attribute Utility Model 

The example cited here is from Shoval and Lugasi 
(Shoval and Lugasi,1987) and is the same situation as given 
in the examples for the additive weight and eigenvector 
models. 

a. Determining Attribute Independence 

Prior to the application of the multi-attribute 
utility model the attribute utility and preference 
independence had to be verified. The utility independence 
was confirmed through the repeated use of the gambling 
technique. In each trial the level of the attributes other 
than i were set to different levels. Utility independence 
was known to be true as U(i') remained constant throughout 
the trials. (Shoval and Lugasi,1987) 

In a similar manner the preference independence 
was also found. Attributes were evaluated taking them two 
at a time. In the first alternative, the first attribute, 
quality of documentation, was set at 2 out of a five-point 
scale and the second attribute, memory capacity, was placed 
at 768K. The second alternative had quality of 
documentation at 4 and memory capacity at i’. All other 
attributes were at their highest respective levels. The 
decision maker was then asked to determine a value for i’ 


where he would be indifferent between the two alternatives. 
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This process was repeated with all of the attributes other 
than quality of documentation and memory capacity set to 


their lowest respective levels. As i‘ was found not to 
change between the two trials, the preference between the 
two attributes was determined to be independent. The rest 
of the attribute pairs were examined in the same way and 
found to be preference independent. (Shoval and 
Lugasi,1987) 

b. Determining Attribute Utility 

In evaluating the utility of the attributes the 

categories were ignored and the 14 attributes were addressed 
in one list. Continuous utility curves were not calculated 
for it was determined that for each attribute only four 
values, one for each alternative, needed to be evaluated. 
To evaluate the memory capacity the decision maker was again 
given two alternatives. Alternative A is certain to receive 
768K capacity. Alternative B is a gamble with the 
probability of p that the capacity will be 1060K and a 
probability of l-p that the capacity will be 576K. The 
minimum acceptable memory capacity having previously been 
defined at 576K and the highest capacity between all 
alternatives being 1060K. After asking the decision maker 
for a value of p where he would be indifferent between the 
two alternatives he responded that p would have to be 0.3 
for him to be indifferent. Thus the memory capacity of 768K 


has the utility value of 0.3. The utility values of all of 
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the 14 attributes were found aetne this same method. The 
utilities found for each of the attributes of each of the 
four alternatives are presented in Table 5. (Shoval and 
Lugasi ,1987) 

c. Determining Attribute Weights 


Attribute weights, k;:, were found as in the 


1’ 
following example for memory capacity. The decision maker 
being provided with two alternative computer systems. One 
has a guaranteed capacity of 1060K and all the remaining 
attributes are at their lowest levels. The other 
alternative is the gamble with a probability of p that all 
attributes are at their highest levels and a probability of 
l-p that all of the attributes are at their lowest levels. 
The decision maker was then asked for a value of p where he 
would feel indifferent between the two alternatives. The 
response was that p would have to be 0.08 for the decision 
maker to be indifferent. Thus the weight of the attribute 
memory capacity is 0.08. The weights for the other 
attributes were found in the same manner and are given in 
Table 5. (Shoval and Lugasi,1987) 
d. Implementing the Model 

As the sum of the weights was greater than 1 the 
multiplicative utility model, Formula 5, was used in this 
example. The constant k was found to be -0.44268 from 
Formula 6. The ranking of the alternatives is given in the 


last line of Table 4. (Shoval and Lugasi,1987) 
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Table 5. Utility Values and Weights 


Alternatives 
Attribute Weight A B Cc 


He HOM HE Hee ee Or O 
POOMDDDDVDDOWOND 
HHH OOOKFKFFHOOOFO 
DPOOIWDDVDODOOWNWWON 
HFOOKOOFPFFPHOOFO 
DOIDOHIDOVOOIMON 
SFPOFPOOOOHOOOHOO 
WODOWNDDWDODDWOUP}Y 


0. 
0. 
0. 
0. 
0. 
0. 
0. 
0. 
0. 
0. 
0. 
0. 
0. 
0. 


Utility .973 .944 .882 .905 
Ranking 1 2 4 3 





3. Advantages & Disadvantages of Multi-Attribute 

Utility 

There are three primary advantages of the multi- 
attribute utility model. First, the model is normative in 
that it is based on the axioms of transitivity and 
continuity which reveal the behavioral rules of the decision 
maker. Secondly, the model verifies attribute independence. 
Finally, the model differs from the additive weight and 
eigenvector models in that it has the capacity to take into 
account the issues of risk and uncertainty. (Shoval and 
Lugasi ,1987) 

The disadvantages of the multi-attribute utility 


model are two fold. First, the decision maker may have 
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difficulty with the gambling technique in making a 
preference between the two alternatives. Secondly, the 
rules of attribute independence, necessary for the 
implementation of the model, are not always met. Thus there 
are situations were the model can not be used. (Shoval and 


Lugasi ,1987) 


D. SUMMARY 

In summary the benefit comparison methods; additive 
weight, eigenvector, and multi-attribute utility, are all 
Similar in nature. All three methods start by defining the 
various advantageous features, or non-cost factors, of an 
information system that are to be evaluated and separate 
those features into distinct attributes. Then each 
attribute is assigned either an importance weight or 
relative utility, dependent on the analysis method. A 
score, or utility, is next found for each alternative by 
attribute. From these scores the various formulas of the 
different methods can then be used to arrive at a final 
ranking of each alternative. The methods differ, of course, 
by the formulas used and if any special conditions have to 
be met, such as attribute independence in the multi- 
attribute utility model. 

1. Advantages & Disadvantages of Benefit Comparison 

All of the methods addressed in this chapter have 

their own merits. The additive weight model has the 
capacity for sensitivity analysis however, in the 
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application of the model in the example, the model does not 
take into account risk or uncertainty. The eigenvector 
model has the capacity to verify a decision maker's 
consistency, yet it too does not consider risk or 
uncertainty. In addition, the eigenvector model overlooks 
the issue of attribute interdependence. The multi-attribute 
utility model has the advantages that it is; normative, it 
examines attribute independence, and it takes into account 
risk and uncertainty. The disadvantage of the multi- 
attribute utility model is that the gambling technique, used 
in the differentiation of alternatives, may be difficult for 
some decision makers. The multi-attribute utility model 
also can not be applied in all situations particularly when 
the rules of attribute independence can not be met. 

2. Benefit Comparison and Non-Cost Factor Analysis 

The different functional and risk factors 

recommended by GSA could be selected as the attributes to be 
evaluated in the application of a benefit comparison method 
for the analysis of the non-cost factors. However, by 
including the functional factor of user acceptance in any 
one of the models would require a scoring technique to 
assess user satisfaction with the proposed alternative. 
None of the benefit comparison methods discussed here 
address how such a score could be determined. Thus there 
remains a need for an analysis method or tool that can be 


applied in the evaluation of user acceptance with respect to 
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the analysis of alternatives. Tools that have been 
recommended for quantifying user acceptance fall under the 
heading of measurements of user satisfaction, which is the 


subject of Chapter V. 
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V. MEASURING USER SATISFACTION 


In the preceding chapter the measurement of user 
satisfaction was introduced as a means of augmentation to 
the benefit comparison methods so that all of the GSA 
stipulated non-cost factors may be evaluated in an analysis 
of alternatives in the acquisition of ADP systems for the 
federal government. The supporters of user satisfaction 
analysis techniques believe so strongly in their work that 
they consider that these techniques should be used more than 
just for augmentation of other evaluation methods 
(Baroudi,Ives,and Olson,1983). These defenders of user 
satisfaction regard the efforts to measure actual 
productivity benefits of information systems as difficult 
and even futile (Baroudi,Ives,and Olson,1983). Others 
insist that when dealing with information systems it is 
impossible to derive useful measures such as return on 
investment or pretax profits (Olson,and Weill,1989). User 
satisfaction, in the view of these authors, is measurable 
and for that fact should be used as the method of choice in 
the analysis of information systems benefit (Baroudi, Ives, 
and Olson,1983). Methods that can be used in the analysis 
of user satisfaction that will be addressed are: the Bailey 
and Pearson tool, the Electre model, and the multi-attribute 


linear value model. 
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A. USER SATISFACTION ANALYSIS METHODS 

Primarily as a substitute to the objective methods of 
information system benefit analysis, such as net present 
value, a subjective measure of user satisfaction has been 
offered. The technique of user satisfaction, also known as 
user information satisfaction, centers around the use of 
Surveys or questionnaires to measure the satisfaction level 
that the users have with their information system. 
Different types of surveys have been suggested, such as 
Single item and multiple item. Surveys of single items or 
functions are of limited utility (Baroudi,Ives, and 
Olson,1983). Multiple item surveys, more broad in scope, 
take a better sample of the users opinions ne the 
information system and therefore dominate as preferred 
measurement techniques of user satisfaction (Baroudi, Ives, 
and Olson,1983). Completed surveys are used to make 
inferences about the beneficial value of an information 
system as a whole. These deductions have been made in the 
past from numerical scales and formulas that have been 
devised by different developers of user satisfaction 


measurement tools. 


B. THE BAILEY AND PEARSON TOOL 

Bailey and Pearson (Bailey and Pearson,1983) present a 
multiple item survey and a respective scoring method to 
measure computer user satisfaction. They identified 39 
different factors of user satisfaction (see Appendix B). 
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Each factor identified was assigned four distinct adjective 
pairs to assess the factor. The documentation factor, for 
instance, was assigned the distinct adjective pairs of: 
- clear vs. hazy 
- available vs. unavailable 
- complete vs. incomplete 
- current vs. obsolete 
In addition to the four distinct pairs of each factor the 
adjective pair satisfactory - unsatisfactory was included as 
a consistency check on the other adjective pairs. Finally 
the adjective pair important - unimportant was included to 
assign weights of importance to the various factors (see 
Appendix B). (Bailey and Pearson, 1983) 
1. Numerical Scales 
To quantify the results of the survey Bailey and 
Pearson (Bailey and Pearson,1983) derived their own 
importance and satisfaction scales. The importance scale, 
used to designate weights of importance to the respective 
factors, was assigned a value range of 0.10 to 1.00 in 0.15 
increments. The values were associated with importance and 
unimportance as shown below: 
1.0 extremely important 
.85 quite 
.70 slightly 
.55 equally 


-40 slightly 


41 


.25 quite 
.10 extremely unimportant 
The satisfaction scale below was used to assign 
values to the first four adjective pairs of each factor 
(Bailey, and Pearson, 1983): 
3 extremely satisfied 
2 quite 
1 slightly 
0 equally 
-1 slightly 
-2 quite 
-3 extremely dissatisfied 
2. Bailey and Pearson's Formula | 
Through the use of the importance and satisfaction 
scales the perception of a user towards the respective 
information system can be captured as a numerical score. 
This value is obtained by first taking a simple arithmetic 
mean of the values I assigned to each of the first 4 
adjective pairs k of each of the factors j using the 
satisfaction scale. Then the summation of the individual 
factorial scores multiplied by their respective weights W, 
from the importance scale, produces the overall measure of 
satisfaction S for each user i as seen in Formula 7. 


(Bailey, and Pearson, 1983) 
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Formula 7 is then normalized to reduce the 
undesirable effects of any factors assessed as neutral or 
meaningless. Any factor is eliminated from consideration if 
all four of the respective distinct adjective pairs were 
evaluated as equal or zero. (Bailey, and Pearson, 1983) 

3. Disadvantages of Bailey and Pearson's tool 

The tool developed by Bailey and Pearson takes a 
broad look at the information system being evaluated and the 
organizational setting in which it was used. However, some 
problems have been identified with the manner in which the 
tool was developed. First, a sample size of only 29 was 
used to validate the tool. Second, each of the 29 subjects 
had participated in the development of the tool (Baroudi, 
Ives and Olson,1983). Baroudi, Ives and Olson (Baroudi, 
Ives and Olson,1983) also found that Bailey and Pearson's 
weighted and unweighted results were highly correlated and 
as such the adjective pair important - unimportant was 
unnecessarily included in the questionnaire. This means 
that questions concerning the importance of individual 
factors did not have a significant impact on the final 
results of the measurement tool. A key problem with the 
tool is that it violates measurement theory by treating 
ordinal measures, the adjective pairs, as interval measures 
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such as distance (Galletta and Lederer,1989). In the 
application of Formula 7 Bailey and Pearson are conducting 
parametric statistical analysis on ordinal measures which is 


a breach measurement theory (Galletta and Lederer,1989). 


C. THE MULTI-ATTRIBUTE LINEAR VALUE PROCEDURE 

A procedure that has been employed in an attempt to 
assess the value of information systems is the multi- 
attribute linear value (MALV) technique (Epstein and 
King,1983). This application of MALV produces more than a 
ranking of alternatives, similar to the Bailey and Pearson 
tool, and though very objective sounding in name this use of 
MALV is in actuality a subjective measure of user 
satisfaction for the data manipulated by the procedure was 
gained through the user survey process (Epstein and 
King,1983). 

1. Description 

The application of the MALV procedure by Epstein and 
King uses a survey, numerical scales, and a formula ina 
Similar manner as Bailey ind. Rearsonee measurement tool of 
user satisfaction. MALV in this application receives data 
through the use of a ten item survey. Each item is a 
factor, or attribute, of information value. All ten 
attributes were selected by the model designer. The ten 
attributes are: (Epstein and King,1983) 
- Reporting cycle 
- Sufficiency 
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- Understandability 

- Freedom from bias 

- Reporting delay 

- Reliability 

- Decision relevance 

- Cost efficiency 

- Comparability 

- Quantitativeness 

2. Epstein and King's Numerical Scales 
Epstein and King then used a scale to attach 

numerical scores to each of the attributes. The scale 
applied to the reporting cycle measures the frequency that 
information inputs have to be generated for user's decision 


making (Epstein and King,1983): 


Reporting Cycle Attribute Scale 


Greatest Satisfaction Possible 100 
Extremely frequent 90 
Fairly infrequent 70 
Moderately frequent 50 
Very frequent 30 
Very infrequent 10 
Lowest Satisfaction Possible 0 


Relative importance weights are obtained by a graphic 
scale on the survey where the response positions are 
stressed and not the scale values. Survey evaluators can 
then quantify the weights by assigning values that match the 
distance ratios between points and the origin. (Epstein and 
King ,1983) 
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3. MALV Formula 
To find the overall value V of an information system 
the multi-attribute linear value model simply multiplies 
each attribute score V(a) by the respective importance 
weight W and adds together all of the results. Formula 8 


summarizes this process. (Epstein,and King,1983) 


10 
v= bo WV (ay) (8) 
i. 


4. Subjectivity in Disguise 

This application of MALV is essentially a subjective 
measure of user satisfaction as the data are acquired 
through the use of a ten item survey. The attribute scales 
and the formula of MALV conflict with measurement theory in 
the same manner as Bailey and Pearson's tool. The model 
fits adjectives to a numerical scale and then analyzes the 
results as if they were interval level measures. The 
validity of the model results are questionable in that they 
are derived from the manipulation of ordinal measures and 
the only legitimate statistical analysis on ordinal measures 
is nonparametric, such as rank ordering (Galletta and 


Lederer ,1989). 
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D. THE ELECTRE MODEL 
1. Description 

A second analysis method of user satisfaction that 
can be used to evaluate completed user satisfaction surveys 
is the Electre model of multiple criteria decision making. 
This model is based on pairwise comparisons of alternatives, 
i and j, and employs two indexes to determine a final 
ranking of various alternatives (Zeleny,1982). The first 
index is the concordance index C(i,j) which is the summation 
of the corresponding criteria weights for all criteria in 
which i out performs j, divided by the summation of all the 
criteria weights of the model (Bui,1981). The second index 
is the discordance index D(i,j) which is found by taking the 
largest difference in evaluation scores for which Jj outranks 
i and dividing that value by the maximum range between 
highest and lowest possible scores (Bui,1981). The two 
indexes are then used in conjunction with threshold values 
selected arbitrarily by the user. The concordance threshold 
P ranges from 0.5 to 1.0 and increases in severity as it 
approaches 1. The discordance threshold Q ranges from 0 to 
1 and increases in severity as it approaches 0. The final 


ranking of alternatives is made as follows (Bui,1981): 


If Then 

C(i,j) >= P and D(i,j) <= Q i outranks j 

C(j,i) >= P and D(j,i) <= Q j outranks i 

1 outranks j and j outranks i i and j are equal 

In all other cases i and j are incomparable 
or the comparison is not 
determinable. 
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2. Example 
a. Applying the Electre Model 
If the Electre model were applied to the 
analysis of user satisfaction for three alternatives, the 
first issue to be settled would be to select the different 
criteria of user satisfaction that would be evaluated in the 
analysis. Along with the criteria, a weight of relative 
importance for each criterion must also be determined. For 
this example the following criteria and weights will be 
used: 
- Understandability of system requests and responses (.2) 
- Ease of learning the new system (.3) 
- Key mapping, or function keys having ena same use as in 
other applications used by the same user (.1) 
- Availability of the system to the user (.1) 
- System response time (.3) 
A second requirement in setting up the Electre 


model is selecting a scale for use in the application of the 


model. For this example the following scale will be used: 
Greatest Satisfaction Possible 100 
Extremely Satisfied 90 
Fairly Satisfied 70 
Moderately Satisfied 50 
Fairly Dissatisfied 30 
Extremely Dissatisfied 10 
Lowest Satisfaction Possible 0 


With the criteria and scale established for the 
application of the model the scores from a user satisfaction 
survey can be put to use in determining a ranking of the 
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various alternatives. Prior to ranking alternatives the 
concordance and discordance thresholds must be selected. 
Here the values of P=.6 and Q=.3 are used for the respective 
thresholds. For this example the scores for the different 


alternatives, given by criteria, are displayed in Table 6. 


Table 6. Sample Criteria Score 


Alternative 
Criteria 2 3 


Understandability 


Ease of learning 
Key mapping 
Availability 
Response time 





b. Ranking Alternatives 

From the example C(1,2) = (.2 +.1 +.3)/l = .6 
and D(1,2) = 20/100 = .2, therefor C(1,2) = P and D(1,2) < Q 
so that alternative one outranks alternative two. Likewise, 
C(1,3) = P and D(1,3) < Q so that alternative one also 
outranks alternative three. In the comparison of 
alternatives two and three, C(2,3) = P and D(2,3) < Q so 
that alternative two also outranks alternative three. The 
final ranking of alternatives is then: 1, 2, and 3. The 
concordance and discordance index figures for this example 


are given in Table 7. 
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Table 7. Concordance and Discordance Figures 


(i,j) C(i,3) D(i,3) 





E. SUMMARY 

User satisfaction analysis is based on user satisfaction 
surveys or questionnaires as illustrated by Bailey and 
Pearson's questionnaire on computer user satisfaction. The 
results of such surveys are interpreted through the use of 
formulas or models such as Bailey and peSeson ts formula, the 
Electre model, or MALV. Prior to the application of these 
formulas the responses to the surveys have to be quantified 
though the use of numerical scales such as the importance 
and satisfaction scales of the Bailey and Pearson tool. 
Measurement tools which may be applied objectively, may be 
subjective measures of user satisfaction if the data the 
tool is based on is obtained through surveys of user 
satisfaction. 

1. Precautions of User Satisfaction Analysis Methods 

Analysts employing measure of user satisfaction 

should be cautious for the results may be undesirable for 
the several reasons. First, the adjective pairs and the 


scales used with them have different meanings to the 
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different respondents of the same questionnaire (Galletta 
and Lederer, 1989). Second, the overall score produced by 
some techniques through the summation of responses to 
specific functions of an information system is invalid 
(Galletta and Lederer,1989). The functions are assumed to 
be homogeneous when in fact they are heterogeneous which 
makes a score based on their summation illogical (Galletta 
and Lederer,1989). If the factors overlap, or are not 
independent of each other, then a tally of the various 
functions simply does not make sense. Overall scores are 
also arrived at without questioning the users’ overall 
perception of the information system which reinforces a 
concern that measures of user satisfaction are unsound 
(Galletta and Lederer,1989). Analysts need to avoid methods 
that violate measurement theory by the mannor in which the 
methods call for parametric statistical analysis on ordinal 
measures such as adjective pairs used in surveys. Lastly, 
officials administering the survey can sway the results by 
influencing the setting or condition in which the survey is 
given (Galletta and Lederer,1989). 
2. Benefits of User Satisfaction Analysis 

Using the Electre model to rank alternatives is a 
legitimate use of ordinal measures. As such the results of 
the Electre model can augment the findings of other analysis 
methods in the analysis of the non-cost factors, 


particularly user acceptance. Thus there is a method that 
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can be used for the analysis of user acceptance as required 
by GSA for the analysis of alternatives in the acquisition 


of federal ADP systems. 


a2 


VI. DISCUSSION AND CONCLUSION 


In the proceeding chapters different approaches to 
benefit analysis were discussed that could be used in the 
analysis of the different non-cost factors that are required 
by the GSA to be evaluated during an analysis of 
alternatives prior to the federal acquisition of an ADP 
system. The two approaches reviewed were benefit comparison 
and user satisfaction. Both of these strategies have their 
own advantages, disadvantages, and possible applications 
within the DoD and the federal government for the evaluation 


of proposed information systems. 


A. BENEFIT COMPARISON 
1. Advantages of Benefit Comparison 
The benefit comparison methods reviewed were: 
additive weight, eigenvector, and multi-attribute utility 
models, and each have some capacity for sensitivity 
analysis. They also have the advantage that they are 
objective methods of benefit analysis as apposed to user 
satisfaction. As mentioned the multi-attribute utility 
model is normative as it is based upon different axioms. 
2. Disadvantages of Benefit Comparison 
A disadvantage of the benefit comparison methods is 
that additive weight and eigenvector models do not have the 
capacity to consider risk or uncertainty when utilized with 
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the attributes given in Table 1. Yet even more critical is 
that these methods can not be used in all situations. As 
addressed, the multi-attribute utility model is incompatible 
with a situation where the attributes are interdependent. 
All of the benefit comparison methods also require that an 
investment of time and effort be made to defining the 
advantages to be gained from the various proposed system so 
that those desired advantages can be used as attributes in 
the analysis. 
3. Feasible Applications of Benefit Comparison 

Benefit comparison techniques are most applicable in 
Situations where the users are willing and able to devote 
the time and expense of defining the awsieea benefits of a 
proposed information system. For a small application that 
can be developed in-house there may not exist the need for a 
detailed benefit analysis. Benefit comparison techniques 
may however, be quite suitable for the evaluation of large 
information systems dealing with administrative activities 
or logistics that are not excluded from the Brooks Act and 
where the GSA recommended non-cost factors must be evaluated 
in the analysis of alternatives for a federal acquisition of 
an ADP system. The GSA functional and risk factors can then 
be used as the attributes in the application of any one of 


the benefit comparison method discussed. 
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B. USER SATISFACTION 
1. Advantages of User Satisfaction 

The main advantage of the user satisfaction is that 
through this technique it is easy to collect data on the 
users’ perceptions of an existing information system. This 
same concept can be applied to gaining information on 
prototypes of proposed information systems. The ease of 
data collection is in reference to the fact that it is 
easier to measure a respondent's answers to a questionnaire 
than finding and proving a cause and effect relationship 
between a system capability and any specific organizational 
benefit. The advantage of easy data collection, used in 
conjunction with such methods as the Electre model, enables 
an analyst to employ user satisfaction to rank order several 
competing alternatives, and to do all of this from a users 
point of view. 

2. Disadvantages of User Satisfaction 

A disadvantage of user satisfaction is that the 
technique has been used incorrectly, as in the Bailey and 
Pearson instrument. Analyst should be cautioned when using 
user Satisfaction not to confuse ordinal measures, such as 
adjective pairs, as interval measures, such as time or 
distance. Analysts should also question methods that score 
alternatives through the summation of interdependent 


attributes. 
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Other disadvantages of the measures of user 
satisfaction include having the responses to the 
questionnaires affected by the manner in which the surveys 
were administered. Inaccurate responses may also be caused 
by what is seen to the respondent's as confusing wording 
used in the questionnaire. Another disadvantage centers 
around the evaluation of proposed system alternatives versus 
evaluating existing systems. Of the literature cited in 
Chapter V on user satisfaction all of the surveys used were 
intended for existing systems. While this technique may be 
selected by some to evaluate system effectiveness, the user 
satisfaction method of benefit analysis appears to be of 
little use in the analysis of proposed imeoemauuten systems 
unless the analysis is tailored so that it can be conducted 
on a prototype of the system. 

3. Feasible Applications of User Satisfaction 

One arena for the application of user satisfaction 
as a technique of evaluating proposed information systems is 
to survey test bed organizations working with prototypes of 
the different alternatives for new proposed system. These 
organizations could receive experience with the different 
prototypes and would be knowledgeable users who could make 
informed responses to surveys on the competing alternatives. 
Using measurements of user satisfaction in a test bed 
organization would be useful for systems that have a high 


degree of user interface, such as in decision support 
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systems or avionic software. The members of these type of 
organizations can offer significant improvements to reduce 
ambiguity through their recommendations concerning: screens 
lay outs, system response messages, or on-line help. Due to 
the large number of units in the military that perform 
Similar actions, this technique could be used to evaluate 
highly interactive systems, from the users perspective, 
prior to a large scale purchase. Again, avionic or some 
weapon system software is a good example where the economies 
of scale for the military could make this type of 
prototyping really pay off. This use of prototyping may 
help preclude the purchase of inappropriate or insufficient 


software. 


C. CONNECTING BENEFITS TO ORGANIZATIONAL GOALS 

Buss (Buss,1983) recommended taking into consideration 
how well an alternative is in line with the organizational 
objectives. This is a vital concern and should be addressed 
by analysts when conducting a benefit analysis. By 
initiating a benefit analysis with a review of 
organizational goals an analyst should be aware of the goals 
that recommendations should address and take into account. 
The organization's goals can also be used as a guide to 
narrow in on what system attributes need to be evaluated 
during the benefit analysis. In Seidmann and Arbel 
(Seidmann and Arbel,1984) a process was presented that 
hierarchically decomposed the problem of defining system 
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attributes. First, the overall organizational objectives 
were reviewed to determine what type of systems are needed. 
In this example a business was examined and it was 
determined that it was in need of an accounting information 
system. Second, the organizational needs were studied in 
respect to the proposed system. This step looked at major 
functions of the planned system, such as accounts 
receivable, payroll, general ledger, and inventory control. 
Third, specific system operations were addressed that were 
required to support the major functions defined earlier. 
Here such operations were identified as file updating and 
report generation. Finally, the system attributes were 
defined that were required to support the specific system 
functions determined in the previous step. Some of the 
attributes identified in this step were: CPU capacity, main 
memory size, compatibility of software, and vendor support. 


(Seidmann and Arbel,1984) 


D. DIFFERENT APPROACHES USED TOGETHER 

There are advantages to using the different approaches 
to benefit analysis in concert with each other and 
capitalizing on the strong points of each. Where system 
capabilities could easily be measured in dollar values, 
present value analysis could be employed to take advantage 
of that method's widely understood methodology. Where 
system benefits are difficult to quantify in dollar term 
benefit comparison methods, such as additive weight, could 
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be used to take advantage of their objective nature and 
capability for sensitivity analysis. The Electre model 
could be used to rank order the alternatives in the analysis 
of user acceptance of a prototype where not only are 
benefits difficult to quantify in dollar term, but where 
analysts need to be cautious with their use of ordinal 
measures in an analysis. By conducting an analysis that 
includes: present value analysis of cost factors, using the 
additive weight model to analyze non-cost factors, and using 
the Electre model to specifically analyze user acceptance, a 
more complete picture can be presented in the analysis of 
alternatives. This will allow for a more informed decision 
in the final selection of alternatives. Both the Eletre 
model and present value analysis could be applied toward 
this type combined analysis without modification. The 
additive weight model would only require slight changes from 


the example given previously. 


E. ADDITIVE WEIGHT MODEL IN FEDERAL ACQUISITIONS 

The functional and risk factor defined by the GSA would 
be used in the application of the additive weight model for 
the analysis of non-cost factors. The functional factor of 
user-acceptance would be excluded from analysis under the 
additive weight model and reserved for the Electre model so 
as to preclude the wrongful use of ordinal measures. The 
break down of non-cost factors in an additive weight 
analysis would resemble Table 8. Starting with these 
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defined attributes and categories an analyst would then have 
to ask the decision maker to assign weights between O and 1 
to each of the attributes and categories. The sum of the 
weights assigned to the two categories would have to equal 1 
and the sum of the weights assigned to the attribute of each 
category would also have to equal 1. Sample weights are 
given in Table 8. 
Table 8 . Sample Weights for Non-Cost Categories and 
Attributes 
Category Attribute Weights 
Zl Functional Factors 
Obsolescence 
Availability 
Reliability 


Maintainability 
Expandability 


Flexibility 
Security 

Privacy 

Personnel Impacts 
Accountability 


Z2 Risk Factors 
Financial 
Technical 
Schedule 





1. Sample Scores for the Additive Weight Model 
The next step in applying the additive weight model 
is to score the alternatives one attribute at a time. A 
range of 0 to 1 is used for the individual attribute scores. 
If the first 13 attribute scores given in Table 2 are used 
aS an example, in combination with the category and 
attribute weights stated in Table 8, then the total weighted 
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score for alternative A would be 0.9272 as expressed in 


Table 9. 


Table 9. Calculation of the Weighted Score for 
Alternative A 


Attribute Category x Attribute x Attribute = Weighted 
Weight Weight Score Score 


0 
QO. 
0 
QO. 
0 
0. 
0. 
QO. 
QO. 
0 
0. 
0. 
0. 
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Total Weighted Score 





2. Advantage of this Application 

The additive weight'’s recognized weakness, that of 
not taking into account risk or uncertainty, is counteracted 
in this application of the model. The weakness is cancel 
because the three different risk factors; financial, 
technical, and schedule, are specifically addressed as 
attributes when using the GSA recommended non-cost factors. 
This application of the additive weight model has the 
advantages of being: conceptually simplistic, calculable on 
any standard spreadsheet application, and in accordance with 
the guidelines set forth by the GSA for federal ADP 
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acquisitions. The results of this application of the 
additive weight model combined together with the results of 
present value analysis and user acceptance analysis, through 
the use of the Electre model, offer a more complete 
evaluation of alternatives than any of the methods could 


offer when used alone. 


F. CONCLUSION 

One method of benefit analysis is not suitable for all 
Situations where proposed information systems are in need of 
evaluation. Nor is one method alone appropriate for the 
evaluation of all of the factors that must be taking into 
account during an analysis of alternative systems. Each 
analysis method has its own advantages and disadvantages. 
Gains can be made by combining methods to take advantage of 
the respective potentials of each. Using different methods 
together would also provide a more complete analysis as some 
methods used alone are inappropriate for certain 
applications, such attempting to use present value analysis 
in an evaluation that includes the measurement of user 
acceptance. System analysts conducting analysis of 
alternatives for a federal acquisition of information 
systems will have to be knowledgeable of the different 
benefit analysis methods. To obtain the required approval 
for development of information systems, and to abide by the 
GSA requirements for analysis of alternatives, system 
analysis will also have to know how to implement the right 
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mix of benefit analysis methods so to fully evaluate both 


cost and non-cost factors. 
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APPENDIX A: EIGENVECTOR MATRICES 


Figure 2. Pairwise Comparison of Attributes by Categories 
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Figure 3. Pairwise Comparison of Categories 
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Figure 4. Comparison of Alternatives by Attributes 
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APPENDIX B: COMPUTER USER SATISFACTION QUESTIONNAIRE 


Use the following scale in response to each pair of 
adjectives: 


fextremelyjquite:;slightlyjequallyislightlyjquitejextremely! 


1. Top management involvement: The positive or negative 
degree of interest, enthusiasm, support, or participation of 
any management level above the user's own level toward 
computer-based information systems or services or toward the 
computer staff which supports them. 

strong | ! ! it weak 
consistent | : | inconsistent 
good t t \ i ] ! ' bad 
Significant | } i insignificant 
satisfactory | ; Lc unsatisfactory 
important | i unimportant 


2. Organizational competition with the EDP unit: The 
contention between the respondent's organizational unit and 
the EDP unit when vying for organizational resources or for 
responsibility for success or failure of computer-based 
information systems or services of interest to both parties. 


productive | | destructive 
rational | : ; | emotional 
low i ; ' ! ' ' high 

harmonious |} | ! | i dissonant 
satisfactory | : | unsatisfactory 
I i I 


important | : (| unimportant 


3. Priorities determination: Policies and procedures which 
establish precedence for the allocation of EDP resources and 
services between different organizational units and their 
requests. 


faner | . unfair 
consistent | ; | inconsistent 
just | : } ~L unjust 
precise | } | tL vague 
satisfactory |} : i unsatisfactory 
important |; : t unimportant 
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4. Charge-back method of payment for services: The 
schedule of charges and the procedures for asSesSSing uSers 
on a pro rata basis for the EDP resources and services that 


they utilize. 


just | : ft unjust 
reasonable | : t unreasonable 
consistent ; } . inconsistent 
known | + unknown 


satisfactory | i unsatisfactory 
important unimportant 


5. Relationship with the EDP staff: The manner and methods 
of interaction, conduct, and association between the user 


and the EDP staff. 


harmonious |} ; | GQissonant 


I 
good 1 0 0 i] ] 0 i bad 
cooperative |} ; i uncooperative 
candid | : | 1 deceitful 
L] I 


satisfactory |} it unsatisfactory 
important |} ! : ! tL unimportant 


6. Communication with the EDP staff: The manner and 
methods of information exchange between the user and the EDP 


staff. 


harmonious | : ; | G@issonant 
productive |} : | destructive 
precise | i vague 
meaningful meaningless 
satisfactory | t unsatisfactory 
important | } ; ( unimportant 


7. Technical competence of the EDP staff: The computer 
technology skills and expertise exhibited by the EDP staff. 


current | : t obsolete 
sufficient | ! : | insufficient 
Superior | | inferior 
high |} ; i Low 
satisfactory | | unsatisfactory 
0 ] ] 


important | ! L unimportant 
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8. Attitude of the EDP staff: 


The willingness and 


commitment of the EDP staff to subjugate external, 
professional goals in favor of organizationally directed 
goals and tasks. 


user-oriented 
cooperative 
courteous 
positive 
satisfactory 
important 


i J I J 
— a on 
a a i 
Se a a 
—— as 1s 
, (aa = "aan ane aaa 


9. Schedule of products and services: The 
timetable for production of information system outputs and 
for provision of computer-based services. 


good 

regular 
reasonable 
acceptable 
satisfactory 
important 


je. | a a a ae a a 2 


ee ee 


; ' : | unreasonable 


gl a LN | eee 


J a a) 


Aine ee ee ee ee ee ae 


10. Time required for new development: Th 
between the user's request for new applications and the 
design, development, and/or implementation 
application systems by the EDP staff. 


short 
dependable 
reasonable 
acceptable 
satisfactory 
important 


Jk |) eee EE EEE 


Ls a Sas ee I 


wl EE ee EE eS ee 


Jl wd a a Ee ES a ee Ses I 


a 2S ee eee ae eee 


A. a | ee | ee 


ll. Processing of change requests: The ma 
required time with which the EDP staff responds to user 

requests for changes in existing computer-based information 
Systems or services. 


fast 

timely 
simple 
flexible 
satisfactory 
important 
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self-centered 
belligerent 
discourteous 
negative 
unsatisfactory 
unimportant 


EDP center 


bad 


irregular 


unacceptable 
unsatisfactory 
unimportant 


e elapsed time 


of the 


long 
undependable 
unreasonable 
unacceptable 
unsatisfactory 
unimportant 


nner, method, and 


slow 

untimely 
complex 

rigid 
unsatisfactory 
unimportant 


12. Vendor support: The type and quality 


rendered by a vendor, 


of the service 


either directly or indirectly, to the 


user to maintain the hardware or software required by that 
organizational status. 


skilled 
sufficient 
eager 
consistent 
satisfactory 
important 


13. Response/turnaround time: 


t t 
———EE——— EE ee ee Eh ae 
I I 
ee 


' I 
i ee oa! ee ee) ee ee 
t I 
ae eS es ee es 
’ I 


os tf 


bungling 
insufficient 
indifferent 
inconsistent 
unsatisfactory 
unimportant 


The elapsed time between a 


user-initiated request for service or action and a reply to 
Response time generally refers to the elapsed 


that request. 
time for terminal type request or entry. 


Turnaround time 


generally refers to the elapsed time for execution of a 
program submitted or requested by a user and the return of 
the output to that user. 


fast 

good 
consistent 
reasonable 
satisfactory 
important 


14. Means of input/output with EDP center: 


' q 1 ! ( 

7, a a 
i ad i i I 
iL, le a ns 
PS er 
a es ae | 


slow 

bad 
inconsistent 
unreasonable 
unsatisfactory 
unimportant 


The method and 


medium by which a user inputs data to and receives output 


from the EDP 


convenient 
clear 
efficient 
organized 
satisfactory 
important 


15. Convenience of access: 


center. 


inconvenient 
hazy 
inefficient 
disorganized 
unsatisfactory 
unimportant 


The ease of difficulty with 


which the user may act to utilize the capability of the 
computer system. 


convenient 
good 

easy 
efficient 
satisfactory 
important 


76 


inconvenient 
bad 

difficult 
inefficient 
unsatisfactory 
unimportant 


L6e. “Hecuracy: 


accurate 
high 
consistent 
sufficient 
satisfactory 
important 


17. Timeliness: 


at a time suitable for its use. 


timely 
reasonable 
consistent 
punctual 
satisfactory 
important 


18. Precision: 


J ee) ee ee a eee 


de"... 


III —E—E— EEE ——E EEE EEE eee 


a ee ees 


alle ee! ae ea eee es ie eee 
j Y I 


Wee ee eS eee eee eee) ee 


from that which it purports to measure. 


sufficient 
consistent 
high 
definite 
satisfactory 
important 


19. Reliability: 


J ' 
2 Se ee eee eee ee 
' ! 


a LL 


Y ' 
SSS ee  _ 
! I 
AS) EE EE ee eee oe 


output information. 


consistent 
high 
superior 
sufficient 
satisfactory 
important 


20. Currency: 


good 

timely 
adequate 
reasonable 
satisfactory 
important 


J Se en eae ee a as Eoceewene os 


Jk ee ES ee ee EE EE Ee a ee 


J re ee ee a es i 


Um 


The correctness of the output information. 


inaccurate 

low 
inconsistent 
insufficient 
unsatisfactory 
unimportant 


The availability of the output information 


untimely 
unreasonable 
inconsistent 
tardy 
unsatisfactory 
unimportant 


The variability of the output information 


insufficient 
inconsistent 
low 

uncertain 
unsatisfactory 
unimportant 


The consistency and dependability of the 


inconsistent 
low 

inferior 
insufficient 
unsatisfactory 
unimportant 


The age of the output information. 


bad 

untimely 
inadequate 
unreasonable 
unsatisfactory 
unimportant 


21. Completeness: The comprehensiveness of the output 
information content. 


incomplete 
consistent inconsistent 
sufficient insufficient 


adequate | i inadequate 


satisfactory |} i unsatisfactory 
important |} ; ; | unimportant 


complete 


22. Format of output: The material design of the layout 
and display of the output contents. 


good |} i bad 
Simple complex 
readable unreadable 
useful !} ; : | useless 


satisfactory ; ' | i unsatisfactory 
important unimportant 


23. Language: The set of vocabulary, syntax, and 
grammatical rules used to interact with the computer 


systems. 


simple } ; | complex 
i 1 } ! ! 


powerful | : : ' | weak 


easy } : } | difficult 
easy-to-use | } i hard-to-use 
! ! | I ! 1 | 


satisfactory ;§ : ' unsatisfactory 


important | : ; | unimportant 


24. Volume of output: The amount of information conveyed 
to a user from computer-based systems. This is expressed 
not only by the number of reports or outputs but also by the 
voluminousness of the output contents. 


concise redundant 
sufficient insufficient 


necessary } ; !' unnecessary 


reasonable ; ; : ' unreasonable 
satisfactory unsatisfactory 


important unimportant 
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25. Relevancy: The degree of congruence between what the 
user wants or requires and what is provided by the 
information products services. 


useful } | useless 
relevant | ( irrelevant 
clear | L hazy 
good } t bad 
satisfactory | iL unsatisfactory 
important | ; i unimportant 


26. Error recovery: The methods and policies governing 
correction and return of system outputs that are incorrect. 


fast | it Slow 


Superior | . inferior 
complete | : : tL incomplete 
Simple } i complex 
satisfactory } ; | unsatisfactory 


important |; ' + unimportant 


27. Security of data: The safeguarding of data from 
misappropriation or unauthorized alteration or loss. 


secure |} it insecure 
good } ; it bad 
definite | } i + uncertain 
complete | ; : ; i incomplete 
satisfactory } - ; (| unsatisfactory 
important !}! ' unimportant 


28. Documentation: The recorded description of an 
information system. This includes formal instructions for 
the utilization of the system. 


clear } | hazy 
available } ( unavailable 
complete | : : 1 incomplete 
current } ; ' obsolete 
satisfactory | - ' unsatisfactory 
important } i unimportant 


iS 


29. Expectat 


pleased |} } ' ; : } 


high 
definite 
optimistic 
satisfactory 
important 


ions: The set of attributes or features of the 
computer-based information products or services that a user 
considers reasonable and due from the computer-based 
information support rendered within the organization. 


i i t 
a a a a 
Sa 
es 
ah 


( 
eS [a ee oe 


30. Understanding of systems: The degree 
that a user possesses about the computer-based information 


system or services that are provided. 


high 
sufficient 
complete 
easy 
satisfactory 
important 


31. Perceived utility: 


displeased 

low 

uncertain 
pessimistic 
unsatisfactory 
unimportant 


of comprehension 


low 
insufficient 
incomplete 
hard 
unsatisfactory 
unimportant 


The user's judgement about the 


relative balance between the cost and the considered 
usefulness of the computer-based information products or 


services that are provided. 
related to providing the resource, 


manpower, and opportunity. The usefulness 
benefits that the user believes to be derived from the 


support. 


high 
positive 
sufficient 
useful 
satisfactory 
important 


32. Confidence in the system: The user's 
certainty about the systems provided. 


assurance or 


high 

strong 
definite 
good 
satisfactory 
important 


The costs include any costs 
including money, time, 


includes any 


low 

negative 
insufficient 
useless 
unsatisfactory 
unimportant 


feelings of 


] j il t 0 i low 
2 S——EEE EEE ES EEE eee) eee 


' ) § 
of |) 


weak 


: H ' uncertain 


i i i i 
| es eS ea Ms betes Po 


bad 


; : | unsatisfactory 


i unimportant 
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33. Feeling 


of participation: The degree 


of involvement 


and commitment which the user shares with the EDP staff and 
others toward the functioning of the computer-based 
information systems and services. 


positive 
encouraged 
sufficient 
involved 
satisfactory 
important 


34. Feeling 


dominate the 


high 
sufficient 
precise 
strong 
satisfactory 
important 


35. Degree of training: 


negative 
repelled 
insufficient 
uninvolved 
unsatisfactory 
unimportant 


of control: The user's awareness of the 
personal power or lack of power to regulate, direct or 
development, alteration, and/or execution of 
the computer-based information systems or services which 
serve the user's perceived function. 


alee sane” Lee Es oe 


low 
insufficient 
vague 

weak 
unsatisfactory 
unimportant 


The amount of specialized 


instruction and practice that is afforded to the user to 
increase the user's proficiency in utilizing the computer 
capability that is unavailable. 


complete 
sufficient 
high 
Superior 
satisfactory 
important 


36. Job effects: 


incomplete 
insufficient 
low 

inferior 
unsatisfactory 
unimportant 


The changes in job freedom and job 


performance that are ascertained by the user as resulting 
from modifications induced by the computer-based information 
system and services. 


liberating 
Significant 
good 
valuable 
satisfactory 
important 
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inhibiting 
insignificant 
bad 

worthless 
unsatisfactory 
unimportant 


37. Organizational position of the EDP function: The 
hierarchical relationship of the EDP function to the overall 
organizational structure. 


appropriate |} } : } t inappropriate 
strong |} H | weak 


clear | : : : i hazy 


progressive |} } } | regressive 
satisfactory |} } : i unsatisfactory 


important | : | unimportant 


38. Flexibility of systems: The capacity of the 
information system to change or to adjust in response to new 
conditions, demands, or circumstances. 


flexible | } | rigid 


versatile | ; | limited 
sufficient | : : : t insufficient 
high | } ' | low 
satisfactory |} ' | unsatisfactory 


important | } tf unimportant 


39. Integration of systems: The ability of systems to 
communicate/transmit data between systems servicing 
different functional areas. 


complete | ! ' incomplete 


sufficient |! } : ' | insufficient 


successful |} 1 tf unsuccessful 


good | | bad 
satisfactory | } | unsatisfactory 


important | ; : t unimportant 


The above user satisfaction questionnaire is basis of a 
copyrighted measurement instrument and is reprinted here 
only as an educational example. Application of the 
measurement instrument, other than in research, should be 
preceded by the receipt of permission from Dr. Pearson. 


(Bailey and Pearson,1983) 
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